v3.5.0: 3.5.0 / 2025-07-14

Compare Source

3.5 is a Long Term Support (LTS) release; see release https://prometheus.io/docs/introduction/release-cycle/
There are a number of new features, and no known breaking changes in this release:

• [FEATURE] PromQL: Add experimental type and unit metadata labels, behind feature flag type-and-unit-labels. #​16228 #​16632 #​16718 #​16743
• [FEATURE] PromQL: Add ts_of_(min|max|last)_over_time, behind feature flag experimental-promql-functions. #​16722 #​16733
• [FEATURE] Scraping: Add global option always_scrape_classic_histograms to scrape a classic histogram even if it is also exposed as native. #​16452
• [FEATURE] OTLP: New config options promote_all_resource_attributes and ignore_resource_attributes. #​16426
• [FEATURE] Discovery: New service discovery for STACKIT Cloud. #​16401
• [ENHANCEMENT] Hetzner SD: Add label_selector to filter servers. #​16512
• [ENHANCEMENT] PromQL: support non-constant parameter in aggregations like quantile and topk. #​16404
• [ENHANCEMENT] UI: Better total target count display when using keep_dropped_targets option. #​16604
• [ENHANCEMENT] UI: Add simple filtering on the /rules page. #​16605
• [ENHANCEMENT] UI: Display query stats in hover tooltip over table query tab. #​16723
• [ENHANCEMENT] UI: Clear search field on /targets page. #​16567
• [ENHANCEMENT] Rules: Check that rules parse without error earlier at startup. #​16601
• [ENHANCEMENT] Promtool: Optional fuzzy float64 comparison in rules unittests. #​16395
• [PERF] PromQL: Reuse histogramStatsIterator where possible. #​16686
• [PERF] PromQL: Reuse storage for custom bucket values for native histograms. #​16565
• [PERF] UI: Optimize memoization and search debouncing on /targets page. #​16589
• [PERF] UI: Fix full-page re-rendering when opening status nav menu. #​16590
• [PERF] Kubernetes SD: use service cache.Indexer to achieve better performance. #​16365
• [PERF] TSDB: Optionally use Direct IO for chunks writing. #​15365
• [PERF] TSDB: When fetching label values, stop work earlier if the limit is reached. #​16158
• [PERF] Labels: Simpler/faster stringlabels encoding. #​16069
• [PERF] Scraping: Reload scrape pools concurrently. #​16595 #​16783
• [BUGFIX] Top-level: Update GOGC before loading TSDB. #​16491
• [BUGFIX] Config: Respect GOGC environment variable if no "runtime" block exists. #​16558
• [BUGFIX] PromQL: Fix native histogram last_over_time. #​16744
• [BUGFIX] PromQL: Fix reported parser position range in errors for aggregations wrapped in ParenExpr #​16041 #​16754
• [BUGFIX] PromQL: Don't emit a value from histogram_fraction or histogram_quantile if classic and native histograms are present at the same timestamp. #​16552
• [BUGFIX] PromQL: Incorrect rounding of [1001ms] to [1s] and similar. #​16478
• [BUGFIX] PromQL: Fix inconsistent / sometimes negative histogram_count and histogram_sum. #​16682
• [BUGFIX] PromQL: Improve handling of NaNs in native histograms. #​16724
• [BUGFIX] PromQL: Fix unary operator precedence in duration expressions. #​16713
• [BUGFIX] PromQL: Improve consistency of avg aggregation and avg_over_time. #​16569 #​16773
• [BUGFIX] UI: Add query warnings and info to graph view. #​16753 #​16759
• [BUGFIX] API: Add HTTP Vary: Origin header to responses to avoid cache poisoning. #​16008
• [BUGFIX] Discovery: Avoid deadlocks by taking locks in consistent order. #​16587
• [BUGFIX] Remote-write: For Azure AD auth, allow empty client_id to suppport system assigned managed identity. #​16421
• [BUGFIX] Scraping: Fix rare memory corruption bug. #​16623
• [BUGFIX] Scraping: continue handling custom-bucket histograms after an exponential histogram is encountered. #​16720
• [BUGFIX] OTLP: Default config not respected when otlp: block is unset. #​16693

v3.4.2: 3.4.2 / 2025-06-26

Compare Source

• [BUGFIX] OTLP receiver: Fix default configuration not being respected if the otlp: block is unset in the config file. #​16693

v3.4.1: 3.4.1 / 2025-05-31

Compare Source

• [BUGFIX] Parser: Add reproducer for a dangling-reference issue in parsers. #​16633

v3.4.0: 3.4.0 / 2025-05-17

Compare Source

• [CHANGE] Config: Make setting out-of-order native histograms feature (--enable-feature=ooo-native-histograms) a no-op. Out-of-order native histograms are now always enabled when out_of_order_time_window is greater than zero and --enable-feature=native-histograms is set. #​16207
• [FEATURE] OTLP translate: Add feature flag for optionally translating OTel explicit bucket histograms into native histograms with custom buckets. #​15850
• [FEATURE] OTLP translate: Add option to receive OTLP metrics without translating names or attributes. #​16441
• [FEATURE] PromQL: allow arithmetic operations in durations in PromQL parser. #​16249
• [FEATURE] OTLP receiver: Add primitive support for ingesting OTLP delta metrics as-is. #​16360
• [ENHANCEMENT] PromQL: histogram_fraction for bucket histograms. #​16095
• [ENHANCEMENT] TSDB: add prometheus_tsdb_wal_replay_unknown_refs_total and prometheus_tsdb_wbl_replay_unknown_refs_total metrics to track unknown series references during WAL/WBL replay. #​16166
• [ENHANCEMENT] Scraping: Add config option for escaping scheme request. #​16066
• [ENHANCEMENT] Config: Add global config option for convert_classic_histograms_to_nhcb. #​16226
• [ENHANCEMENT] Alerting: make batch size configurable (--alertmanager.notification-batch-size). #​16254
• [PERF] Kubernetes SD: make endpointSlice discovery more efficient. #​16433
• [BUGFIX] Config: Fix auto-reload on changes to rule and scrape config files. #​16340
• [BUGFIX] Scraping: Skip native histogram series if ingestion is disabled. #​16218
• [BUGFIX] TSDB: Handle metadata/tombstones/exemplars for duplicate series during WAL replay. #​16231
• [BUGFIX] TSDB: Avoid processing exemplars outside the valid time range during WAL replay. #​16242
• [BUGFIX] Promtool: Add feature flags for PromQL features. #​16443
• [BUGFIX] Rules: correct logging of alert name & template data. #​15093
• [BUGFIX] PromQL: Use arithmetic mean for histogram_stddev() and histogram_stdvar() . #​16444

v3.3.1: 3.3.1 / 2025-05-02

Compare Source

• [BUGFIX] Azure SD: Fix panic on malformed log message. #​16434 #​16210
• [BUGFIX] Config: Update GOGC before loading TSDB. #​16491

v3.3.0: 3.3.0 / 2025-04-15

Compare Source

• [FEATURE] PromQL: Implement idelta() and irate() for native histograms. #​15853
• [ENHANCEMENT] Scaleway SD: Add __meta_scaleway_instance_public_ipv4_addresses and __meta_scaleway_instance_public_ipv6_addresses labels. #​14228
• [ENHANCEMENT] TSDB: Reduce locking while reloading blocks. #​12920
• [ENHANCEMENT] PromQL: Allow UTF-8 labels in label_replace(). #​15974
• [ENHANCEMENT] Promtool: tsdb create-blocks-from openmetrics can now read from a Pipe. #​16011
• [ENHANCEMENT] Rules: Add support for anchors and aliases in rule files. #​14957
• [ENHANCEMENT] Dockerfile: Make /prometheus writable. #​16073
• [ENHANCEMENT] API: Include scrape pool name for dropped targets in /api/v1/targets. #​16085
• [ENHANCEMENT] UI: Improve time formatting and copying of selectors. #​15999 #​16165
• [ENHANCEMENT] UI: Bring back vertical grid lines and graph legend series toggling instructions. #​16163 #​16164
• [ENHANCEMENT] Mixin: The cluster label can be customized using clusterLabel. #​15826
• [PERF] TSDB: Optimize some operations on head chunks by taking shortcuts. #​12659
• [PERF] TSDB & Agent: Reduce memory footprint during WL replay. #​15778
• [PERF] Remote-Write: Reduce memory footprint during WAL replay. #​16197
• [PERF] API: Reduce memory footprint during header parsing. #​16001
• [PERF] Rules: Improve dependency evaluation, enabling better concurrency. #​16039
• [PERF] Scraping: Improve scraping performance for native histograms. #​15731
• [PERF] Scraping: Improve parsing of created timestamps. #​16072
• [BUGFIX] Scraping: Bump cache iteration after error to avoid false duplicate detections. #​16174
• [BUGFIX] Scraping: Skip native histograms series when ingestion is disabled. #​16218
• [BUGFIX] PromQL: Fix counter reset detection for native histograms. #​15902 #​15987
• [BUGFIX] PromQL: Fix inconsistent behavior with an empty range. #​15970
• [BUGFIX] PromQL: Fix inconsistent annotation in quantile_over_time(). #​16018
• [BUGFIX] PromQL: Prevent label_join() from producing duplicates. #​15975
• [BUGFIX] PromQL: Ignore native histograms in scalar(), sort() and sort_desc(). #​15964
• [BUGFIX] PromQL: Fix annotations for binary operations between incompatible native histograms. #​15895
• [BUGFIX] Alerting: Consider alert relabeling when deciding whether alerts are dropped. #​15979
• [BUGFIX] Config: Set GoGC to the default value in case of an empty configuration. #​16052
• [BUGFIX] TSDB: Fix unknown series errors and potential data loss during WAL replay when inactive series are removed from the head and reappear before the next WAL checkpoint. #​16060
• [BUGFIX] Scaleway SD: The public IP will no longer be set to __meta_meta_scaleway_instance_public_ipv4 if it is an IPv6 address. #​14228
• [BUGFIX] UI: Display the correct value of Alerting rules' keep_firing_for. #​16211

v3.2.1: 3.2.1 / 2025-02-25

Compare Source

• [BUGFIX] Don't send Accept header escape=allow-utf-8 when metric_name_validation_scheme: legacy is configured. #​16061

v3.2.0: 3.2.0 / 2025-02-17

Compare Source

Prometheus v3.2.0 Changelog

• [CHANGE] relabel: Replace actions can now use UTF-8 characters in targetLabel field. Note that $<chars> or ${<chars>} will be expanded. This also apply to replacement field for LabelMap action. #​15851
• [CHANGE] rulefmt: Rule names can use UTF-8 characters, except { and } characters (due to common mistake checks). #​15851
• [FEATURE] remote/otlp: Add feature flag otlp-deltatocumulative to support conversion from delta to cumulative. #​15165
• [ENHANCEMENT] openstack SD: Discover Octavia loadbalancers. #​15539
• [ENHANCEMENT] scrape: Add metadata for automatic metrics to WAL for metadata-wal-records feature. #​15837
• [ENHANCEMENT] promtool: Support linting of scrape interval, through lint option too-long-scrape-interval. #​15719
• [ENHANCEMENT] promtool: Add --ignore-unknown-fields option. #​15706
• [ENHANCEMENT] ui: Make "hide empty rules" and hide empty rules" persistent #​15807
• [ENHANCEMENT] web/api: Add a limit parameter to /query and /query_range. #​15552
• [ENHANCEMENT] api: Add fields Node and ServerTime to /status. #​15784
• [PERF] Scraping: defer computing labels for dropped targets until they are needed by the UI. #​15261
• [BUGFIX] remotewrite2: Fix invalid metadata bug for metrics without metadata. #​15829
• [BUGFIX] remotewrite2: Fix the unit field propagation. #​15825
• [BUGFIX] scrape: Fix WAL metadata for histograms and summaries. #​15832
• [BUGFIX] ui: Merge duplicate "Alerts page settings" sections. #​15810
• [BUGFIX] PromQL: Fix <aggr_over_time> functions with histograms. #​15711

v3.1.0: 3.1.0 / 2025-01-02

Compare Source

What's Changed

• [SECURITY] upgrade golang.org/x/crypto to address reported CVE-2024-45337. #​15691
• [CHANGE] Notifier: Increment prometheus_notifications_errors_total by the number of affected alerts rather than per batch. #​15428
• [CHANGE] API: list rules field "groupNextToken:omitempty" renamed to "groupNextToken". #​15400
• [ENHANCEMENT] OTLP translate: keep identifying attributes in target_info. #​15448
• [ENHANCEMENT] Paginate rule groups, add infinite scroll to rules within groups. #​15677
• [ENHANCEMENT] TSDB: Improve calculation of space used by labels. #​13880
• [ENHANCEMENT] Rules: new metric rule_group_last_rule_duration_sum_seconds. #​15672
• [ENHANCEMENT] Observability: Export 'go_sync_mutex_wait_total_seconds_total' metric. #​15339
• [ENHANCEMEN] Remote-Write: optionally use a DNS resolver that picks a random IP. #​15329
• [PERF] Optimize l=~".+" matcher. #​15474, #​15684
• [PERF] TSDB: Cache all symbols for compaction . #​15455
• [PERF] TSDB: MemPostings: keep a map of label values slices. #​15426
• [PERF] Remote-Write: Remove interning hook. #​15456
• [PERF] Scrape: optimize string manipulation for experimental native histograms with custom buckets. #​15453
• [PERF] TSDB: reduce memory allocations. #​15465, #​15427
• [PERF] Storage: Implement limit in mergeGenericQuerier. #​14489
• [PERF] TSDB: Optimize inverse matching. #​14144
• [PERF] Regex: use stack memory for lowercase copy of string. #​15210
• [PERF] TSDB: When deleting from postings index, pause to unlock and let readers read. #​15242
• [BUGFIX] Main: Avoid possible segfault at exit. (#​15724)
• [BUGFIX] Rules: Do not run rules concurrently if uncertain about dependencies. #​15560
• [BUGFIX] PromQL: Adds test for absent, absent_over_time and deriv func with histograms. #​15667
• [BUGFIX] PromQL: Fix various bugs related to quoting UTF-8 characters. #​15531
• [BUGFIX] Scrape: fix nil panic after scrape loop reload. #​15563
• [BUGFIX] Remote-write: fix panic on repeated log message. #​15562
• [BUGFIX] Scrape: reload would ignore always_scrape_classic_histograms and convert_classic_histograms_to_nhcb configs. #​15489
• [BUGFIX] TSDB: fix data corruption in experimental native histograms. #​15482
• [BUGFIX] PromQL: Ignore histograms in all time related functions. #​15479
• [BUGFIX] OTLP receiver: Convert metric metadata. #​15416
• [BUGFIX] PromQL: Fix resets function for histograms. #​15527
• [BUGFIX] PromQL: Fix behaviour of changes() for mix of histograms and floats. #​15469
• [BUGFIX] PromQL: Fix behaviour of some aggregations with histograms. #​15432
• [BUGFIX] allow quoted exemplar keys in openmetrics text format. #​15260
• [BUGFIX] TSDB: fixes for rare conditions when loading write-behind-log (WBL). #​15380
• [BUGFIX] round() function did not remove __name__ label. #​15250
• [BUGFIX] Promtool: analyze block shows metric name with 0 cardinality. #​15438
• [BUGFIX] PromQL: Fix count_values for histograms. #​15422
• [BUGFIX] PromQL: fix issues with comparison binary operations with bool modifier and native histograms. #​15413
• [BUGFIX] PromQL: fix incorrect "native histogram ignored in aggregation" annotations. #​15414
• [BUGFIX] PromQL: Corrects the behaviour of some operator and aggregators with Native Histograms. #​15245
• [BUGFIX] TSDB: Always return unknown hint for first sample in non-gauge histogram chunk. #​15343
• [BUGFIX] PromQL: Clamp functions: Ignore any points with native histograms. #​15169
• [BUGFIX] TSDB: Fix race on stale values in headAppender. #​15322
• [BUGFIX] UI: Fix selector / series formatting for empty metric names. #​15340
• [BUGFIX] OTLP receiver: Allow colons in non-standard units. #​15710

v3.0.1: 3.0.1 / 2024-11-28

Compare Source

The first bug fix release for Prometheus 3.

• [BUGFIX] Promql: Make subqueries left open. #​15431
• [BUGFIX] Fix memory leak when query log is enabled. #​15434
• [BUGFIX] Support utf8 names on /v1/label/:name/values endpoint. #​15399

v3.0.0: 3.0.0 / 2024-11-14

Compare Source

This release includes new features such as a brand new UI and UTF-8 support enabled by default. As this marks the first new major version in seven years, several breaking changes are introduced. The breaking changes are mainly around the removal of deprecated feature flags and CLI arguments, and the full list can be found below. For users that want to upgrade we recommend to read through our migration guide.

• [CHANGE] Set the GOMAXPROCS variable automatically to match the Linux CPU quota. Use --no-auto-gomaxprocs to disable it. The auto-gomaxprocs feature flag was removed. #​15376
• [CHANGE] Set the GOMEMLIMIT variable automatically to match the Linux container memory limit. Use --no-auto-gomemlimit to disable it. The auto-gomemlimit feature flag was removed. #​15373
• [CHANGE] Scraping: Remove implicit fallback to the Prometheus text format in case of invalid/missing Content-Type and fail the scrape instead. Add ability to specify a fallback_scrape_protocol in the scrape config. #​15136
• [CHANGE] Remote-write: default enable_http2 to false. #​15219
• [CHANGE] Scraping: normalize "le" and "quantile" label values upon ingestion. #​15164
• [CHANGE] Scraping: config scrape_classic_histograms was renamed to always_scrape_classic_histograms. #​15178
• [CHANGE] Config: remove expand-external-labels flag, expand external labels env vars by default. #​14657
• [CHANGE] Disallow configuring AM with the v1 api. #​13883
• [CHANGE] regexp . now matches all characters (performance improvement). #​14505
• [CHANGE] holt_winters is now called double_exponential_smoothing and moves behind the experimental-promql-functions feature flag. #​14930
• [CHANGE] API: The OTLP receiver endpoint can now be enabled using --web.enable-otlp-receiver instead of --enable-feature=otlp-write-receiver. #​14894
• [CHANGE] Prometheus will not add or remove port numbers from the target address. no-default-scrape-port feature flag removed. #​14160
• [CHANGE] Logging: the format of log lines has changed a little, along with the adoption of Go's Structured Logging package. #​14906
• [CHANGE] Don't create extra _created timeseries if feature-flag created-timestamp-zero-ingestion is enabled. #​14738
• [CHANGE] Float literals and time durations being the same is now a stable fetaure. #​15111
• [CHANGE] UI: The old web UI has been replaced by a completely new one that is less cluttered and adds a few new features (PromLens-style tree view, better metrics explorer, "Explain" tab). However, it is still missing some features of the old UI (notably, exemplar display and heatmaps). To switch back to the old UI, you can use the feature flag --enable-feature=old-ui for the time being. #​14872
• [CHANGE] PromQL: Range selectors and the lookback delta are now left-open, i.e. a sample coinciding with the lower time limit is excluded rather than included. #​13904
• [CHANGE] Kubernetes SD: Remove support for discovery.k8s.io/v1beta1 API version of EndpointSlice. This version is no longer served as of Kubernetes v1.25. #​14365
• [CHANGE] Kubernetes SD: Remove support for networking.k8s.io/v1beta1 API version of Ingress. This version is no longer served as of Kubernetes v1.22. #​14365
• [CHANGE] UTF-8: Enable UTF-8 support by default. Prometheus now allows all UTF-8 characters in metric and label names. The corresponding utf8-name feature flag has been removed. #​14705
• [CHANGE] Console: Remove example files for the console feature. Users can continue using the console feature by supplying their own JavaScript and templates. #​14807
• [CHANGE] SD: Enable the new service discovery manager by default. This SD manager does not restart unchanged discoveries upon reloading. This makes reloads faster and reduces pressure on service discoveries' sources. The corresponding new-service-discovery-manager feature flag has been removed. #​14770
• [CHANGE] Agent mode has been promoted to stable. The feature flag agent has been removed. To run Prometheus in Agent mode, use the new --agent cmdline arg instead. #​14747
• [CHANGE] Remove deprecated remote-write-receiver,promql-at-modifier, and promql-negative-offset feature flags. #​13456, #​14526
• [CHANGE] Remove deprecated storage.tsdb.allow-overlapping-blocks, alertmanager.timeout, and storage.tsdb.retention flags. #​14640, #​14643
• [FEATURE] OTLP receiver: Ability to skip UTF-8 normalization using otlp.translation_strategy = NoUTF8EscapingWithSuffixes configuration option. #​15384
• [FEATURE] Support config reload automatically - feature flag auto-reload-config. #​14769
• [ENHANCEMENT] Scraping, rules: handle targets reappearing, or rules moving group, when out-of-order is enabled. #​14710
• [ENHANCEMENT] Tools: add debug printouts to promtool rules unit testing #​15196
• [ENHANCEMENT] Scraping: support Created-Timestamp feature on native histograms. #​14694
• [ENHANCEMENT] UI: Many fixes and improvements. #​14898, #​14899, #​14907, #​14908, #​14912, #​14913, #​14914, #​14931, #​14940, #​14945, #​14946, #​14972, #​14981, #​14982, #​14994, #​15096
• [ENHANCEMENT] UI: Web UI now displays notifications, e.g. when starting up and shutting down. #​15082
• [ENHANCEMENT] PromQL: Introduce exponential interpolation for native histograms. #​14677
• [ENHANCEMENT] TSDB: Add support for ingestion of out-of-order native histogram samples. #​14850, #​14546
• [ENHANCEMENT] Alerts: remove metrics for removed Alertmanagers. #​13909
• [ENHANCEMENT] Kubernetes SD: Support sidecar containers in endpoint discovery. #​14929
• [ENHANCEMENT] Consul SD: Support catalog filters. #​11224
• [ENHANCEMENT] Move AM discovery page from "Monitoring status" to "Server status". #​14875
• [PERF] TSDB: Parallelize deletion of postings after head compaction. #​14975
• [PERF] TSDB: Chunk encoding: shorten some write sequences. #​14932
• [PERF] TSDB: Grow postings by doubling. #​14721
• [PERF] Relabeling: Optimize adding a constant label pair. #​12180
• [BUGFIX] UI: fix selector / series formatting for empty metric names. #​15341
• [BUGFIX] PromQL: Fix stddev+stdvar aggregations to always ignore native histograms. #​14941
• [BUGFIX] PromQL: Fix stddev+stdvar aggregations to treat Infinity consistently. #​14941
• [BUGFIX] OTLP receiver: Preserve colons when generating metric names in suffix adding mode (this mode is always enabled, unless one uses Prometheus as a library). #​15251
• [BUGFIX] Scraping: Unit was missing when using protobuf format. #​15095
• [BUGFIX] PromQL: Only return "possible non-counter" annotation when rate returns points. #​14910
• [BUGFIX] TSDB: Chunks could have one unnecessary zero byte at the end. #​14854
• [BUGFIX] "superfluous response.WriteHeader call" messages in log. #​14884
• [BUGFIX] PromQL: Unary negation of native histograms. #​14821
• [BUGFIX] PromQL: Handle stale marker in native histogram series (e.g. if series goes away and comes back). #​15025
• [BUGFIX] Autoreload: Reload invalid yaml files. #​14947
• [BUGFIX] Scrape: Do not override target parameter labels with config params. #​11029

Full Changelog: prometheus/prometheus@v2.55.0...v3.0.0